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Section  I:  Project  Summary 


1 .  Overview  of  Proj  ect 

This  project  is  performed  under  the  Office  of  Naval  Research  program  on  Basic  and  Applied  Research  in 
Sea-Based  Aviation  (ONR  BAA12-SN-0028).  This  project  addresses  the  Sea  Based  Aviation  (SBA) 
virtual  dynamic  interface  (VDI)  research  topic  area  “Fast,  high-fidelity  physics-based  simulation  of 
coupled  aerodynamics  of  moving  ship  and  maneuvering  rotorcraff The  work  is  a  collaborative  effort 
between  Penn  State,  NAVAIR,  and  Combustion  Research  and  Flow  Technology  (CRAFT  Tech).  This 
document  presents  progress  at  Penn  State  University. 

All  software  supporting  piloted  simulations  must  run  at  real  time  speeds  or  faster.  This  requirement 
drives  the  number  of  equations  that  can  be  solved  and  in  turn  the  fidelity  of  supporting  physics  based 
models.  For  real-time  aircraft  simulations,  all  aerodynamic  related  information  for  both  the  aircraft  and 
the  environment  are  incorporated  into  the  simulation  by  way  of  lookup  tables.  This  approach  decouples 
the  aerodynamics  of  the  aircraft  from  the  rest  of  its  external  environment.  For  example,  ship  airwake  are 
calculated  using  CFD  solutions  without  the  presence  of  the  helicopter  main  rotor.  The  gusts  from  the 
turbulent  ship  airwake  are  then  re -played  into  the  aircraft  aerodynamic  model  via  look-up  tables.  For  up 
and  away  simulations,  this  approach  works  well.  However,  when  an  aircraft  is  flying  very  close  to 
another  body  (i.e.  a  ship  superstructure)  significant  aerodynamic  coupling  can  exist.  The  main  rotor  of 
the  helicopter  distorts  the  flow  around  the  ship  possibly  resulting  significant  differences  in  the 
disturbance  on  the  helicopter.  In  such  cases  it  is  necessary  to  perform  simultaneous  calculations  of  both 
the  Navier-Stokes  equations  and  the  aircraft  equations  of  motion  in  order  to  achieve  a  high  level  of 
fidelity.  This  project  will  explore  novel  numerical  modeling  and  computer  hardware  approaches  with 
the  goal  of  real  time,  fully  coupled  CFD  for  virtual  dynamic  interface  modeling  &  simulation. 

Penn  State  is  supporting  the  project  through  integration  of  their  GENHEL-PSU  simulation  model  of  a 
utility  helicopter  with  CRAFT  Tech’s  flow  solvers.  Penn  State  will  provide  their  piloted  simulation 
facility  (the  VLRCOE  rotorcraft  simulator)  for  preliminary  demonstrations  of  pilot-in-the-loop 
simulations.  Finally,  Penn  State  will  provide  support  for  a  final  demonstration  of  the  methods  on  the 
NAVAIR  Manned  Flight  Simulator. 

Activities  this  period 

During  this  reporting  period,  PSU  continued  to  collaborate  with  CRAFT-Tech  to  improve  the  efficiency 
of  the  new  coupling  interface.  We  have  demonstrated  our  first  real-time  pilot-in-the-loop  simulation  for 
a  simplified  CFD  wake  case,  as  well  as  a  near-real-time/real-time  batch  simulations  with  a  pilot  model 
and  a  coarse  and  fine  computational  grid,  respectively.  These  simulations  were  run  with  the  MPMD-MPI 
communication  protocol  and  using  the  CRAFT  structured  flow  solver. 

Demonstrations  of  Real-time  Fully  Coupled  Simulations 

Several  efforts  have  been  conducted  towards  the  real-time  Pilot-in-the-loop  (PIL)  CFD  simulations.  All 
these  efforts  have  been  performed  on  CRAFT  Tech’s  in-house  cluster  with  32  nodes,  each  containing  8 
Intel  Xeon  E5530  processors  (2.4  GHz),  using  a  40Gbps  Infiniband  interconnect.  (32  nodes  x  8  procs  = 
256  processors) 


Table  1  -  Description  of  simulated  cases 


Case  # 

Piloted  /  Non-piloted? 

Flow  Assumption 

Grid  Size 

Case  1 

Non-piloted 

Viscous  flow 

1 .2  million 

Case  2 

Non-piloted 

Inviscid  flow 

380k 

Case  3 

Piloted 

Inviscid  flow 

380k 

Table  1  shows  the  properties  of  the  simulation  cases  performed.  A  set  of  three  simple  shedding  small 
backstep  cases  were  performed  using  two  relatively  small  computational  domains:  1)  a  finer  grid  with  1.2 
million  cells,  and  2)  a  coarser  grid  with  360k  cells.  Figure  1  shows  the  comparisons  of  the  grid 
resolutions  used  in  the  demonstration  cases.  The  minimum  cell  size  in  the  refined  region  downstream  of 
the  backstep  was  1ft  and  4ft  for  Case  1  and  Case  2,  respectively. 

Case  I:  Non-real-time  Baseline  Coupled  Simulation 

The  first  case  was  chosen  as  a  “baseline”  case,  which  was  expected  to  generate  a  better-resolved  shedding 
wake.  The  structured  CRAFT  CFD  solver  was  used  with  viscous  flux  term  is  included  and  a  wall  function 
boundary  applied  to  the  backstep  surfaces.  A  relatively  small  computational  domain  with  1.2  million  cells 
was  used  in  the  simulation.  Using  the  NLDI  controller,  the  helicopter  executed  a  landing  trajectory  within 
a  simulation  period  of  60  seconds.  The  GENHEL-PSU  time  step  was  0.01  seconds  and  the  CFD  time  step 
was  0.005  seconds,  which  took  0.075  seconds  per  iteration  to  run  on  256  processors  (15  times  slower  than 
real-time).  GENHEL-PSU  was  set  to  run  one  time  step  and  CFD  solver  was  set  to  run  two  time  steps  in 
between  the  data  exchanges.  The  dimensions  of  the  backstep  (width  =  60  ft)  produce  shedding 
characteristics  similar  to  a  ship  flight  deck  for  this  case. 


Figure  1.  Comparison  of  grid  resolutions  of  the  computational  domains  used  in  the  demonstration  cases. 
Case  II:  Real-time  Coupled  Simulation 

Using  the  CRAFT  CFD  structured  solver  running  with  5th-order  upwind  Roe/TVD  scheme  without 
viscous  flux  terms  on  a  structured  grid  (380k  cells),  a  real-time  coupled  simulation  was  achieved.  The 
same  NLDI  controller  was  used  to  fly  the  helicopter  on  the  same  landing  trajectory;  in  order  to  compare 
the  results  with  the  more  resolved  wake  solution  of  Case  I.  In  this  case,  using  coarser  mesh  structure  and 


inviscid  flow  assumption  allowed  us  to  use  a  bigger  time  step  in  the  CFD  calculations.  The  CFD  time 
step  was  0.02  seconds  and  GENHEL-PSU  exchanged  data  with  CRAFT  CFD  every  2  steps.  The  CFD 
calculations  took  about  0.019  seconds  per  iteration  to  run  a  time  step  of  0.02  seconds,  resulting  in  real¬ 
time  execution  speeds.  The  shedding  characteristics  were  not  as  well  resolved  as  the  non-real-time  CFD 
case  (Case  1)  because  of  the  inviscid  flow  assumption  and  the  coarser  grid  resolution. 


Figure  2  shows  a  comparison  of  the  streamline  distributions  between  Casel  and  Case  2.  In  Case  1,  the 
recirculation  region  behind  the  backstep  is  narrower  compared  to  Case  2.  The  streamline  distribution  is 
more  laminar  compared  to  Case  1.  Figure  3  shows  the  vorticity  distribution  of  Case  1  and  Case  2  at  the 
same  simulation  time  indicating  the  more  resolved  shedding  flowfield  of  the  backstep  wake  and  the  less 
dissipative  rotor  tip  wake  of  Case  1  due  to  the  finer  grid  resolution.  The  inviscid  flow  assumption  and  the 
coarser  grid  resolution  of  Case  2  results  in  relatively  poor  shedding  characteristics  compared  to  Case  1 . 
However,  these  simplifications  were  required  to  achieve  the  real-time  execution  speeds  needed  for  a 
demonstration  of  the  pilot-in-the-loop  simulation  framework  on  available  computing  hardware. 


Figure  2.  Streamline  distributions  for  Case  1  and  Case  2. 


Figure  3.  Vorticity  distribution  for  Case  1  and  Case  2. 

Figure  4  shows  the  dynamic  response  of  the  helicopter  as  it  executes  the  landing  maneuver  in  the 
simulated  cases.  As  mentioned  before,  the  NLDI  autopilot  controller  holds  the  helicopter  at  the  desired 
trajectory.  In  the  “No-coupling”  case,  GENHEL-PSU  uses  the  Pitt-Peters  inflow  model  to  calculate  the 
rotor  downwash  and  there  is  no  airwake  disturbance  to  the  helicopter.  However,  on  Case  1  and  Case  2, 
the  GENHEL-PSU  code  is  fully  coupled  with  the  CFD  solver  and  the  Pitt-Peters  inflow  model  is  replaced 
with  the  CFD-calculated  flowfield  solution.  As  shown  in  Figure  4a,  the  controller  successfully  holds  the 
position  of  the  helicopter  on  the  desired  trajectory.  The  helicopter  drifts  more  in  Case  1  compared  to  the 
other  two  cases.  A  similar  behavior  can  also  be  seen  on  the  attitude  response  of  the  helicopter.  There  are 


many  fluctuations  on  the  roll  and  pitch  dynamics  of  the  helicopter  in  Case  1.  We  see  a  smoother  flight  on 
‘Case  2’  and  ‘No  Coupling’  cases.  This  is  a  result  of  inviscid  flow  assumption  and  the  less  resolved,  more 
dissipative  grid,  which  suppresses  the  turbulent  behavior  of  the  flow.  Additionally,  the  helicopter  trims  at 
slightly  higher  collective  input  on  Case  1  when  it  gets  close  to  the  ground,  since  the  ground  effect  is 
modeled  more  precisely  in  this  case. 
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Figure  4.  Variations  in  dynamics  of  the  simulated  helicopter  approaching  to  the  small  backstep. 

Case  III:  Real-time  Pilot-in-the-Loop  CFD  Simulation 

For  the  Pilot-in-the-Loop  CFD  (PILCFD)  demonstration  cases,  a  network  configuration  on  a  CRAFT 
Tech  Linux  cluster  was  developed  as  shown  in  Figure  5.  The  helicopter  flight  dynamics  simulation 
(GENHEL-PSU)  was  run  on  the  head  node  of  the  cluster,  which  enabled  network  communication  ports 
outside  the  cluster  network  to  a  Linux  workstation  running  the  joystick  controller  (for  pilot  inputs)  and  X- 
Plane  (Figure  6)  graphics  to  display  the  simulation.  The  CRAFT  CFD  solver  was  run  in  parallel  over  all 
32  compute  nodes  (256  processors)  on  the  cluster  using  a  40Gbps  Infiniband  interconnect. 


LINUX  WORKSTATION  LINUX  CLUSTER 


Figure  5.  PILCFD  demonstration  case  network  configuration. 

Using  the  same  grid  as  in  Case  2  (380k),  with  a  person  flying  the  helicopter  in  the  wake,  a  real-time 
piloted  simulation  (Figure  7)  was  performed.  The  CFD  calculations  (flow  calculations  +  MPI  data 
exchange)  took  about  0.02s  to  run  a  time  step  of  0.02  seconds.  Therefore  GENHEL  (0.01s  time  step) 
exchanged  data  with  the  CFD  every  2  steps.  The  joystick  controller  was  set  to  0.025s  update  interval 
which  gave  a  smooth  response  of  the  flight  dynamics  with  the  simulation.  It  was  found  that  this  interval 
must  be  slightly  greater  or  equal  to  the  physical  wall  time  required  to  complete  one  CFD  solver  time  step 
plus  the  time  required  to  complete  rest  of  the  operations  (time  required  to  complete  two  GENHEL  time 
step  +  time  required  to  complete  all  the  data  transfer  over  the  network  +  network  latency).  This  update 
interval  produced  a  smooth  flight  simulation  with  the  dynamics  and  CFD  working  in  ’’real-time” 
execution  speeds. 


Figure  6  -  A  representative  frigate  ship  and  UH-60  helicopter  model  used  for  the  PILCFD  demonstrations. 


Figure  7  -  Real-time  PILCFD  demonstration  case. 


The  same  simulation  case  was  repeated  with  CRAFT  CFD  writing  the  solution  data  to  disk  during  the 
simulation.  This  operation  significantly  slowed  the  simulation  down  (it  took  0.045sec  to  run  a  time  step 
of  0.02  seconds)  but  enabled  us  to  generate  the  CFD  data  that  is  synched  to  the  X-Plane  graphics 
animation  as  well  as  the  flight  dynamics  data  from  GENHEL-PSU.  The  joystick  was  set  to  a  0.05s 
update  interval,  which  produced  a  smooth  flight  simulation  with  the  dynamics  and  CFD  working  in 
“slow  motion”,  about  2.5  times  slower  than  the  real-time.  The  obtained  CFD  solution  was  not  as 
accurate  as  Casel  on  this  coarser  grid,  however  it  successfully  demonstrated  the  simulation  framework 
with  flow  calculations  based  on  the  pilot’s  inputs  to  steer  the  helicopter. 

The  timings  achieved  on  all  three  cases  are  listed  on  Table  2.  On  the  last  two  cases  (Case  2  and  3),  “real- 
time”  execution  speeds  were  achieved. 


Table  2.  Timings  achieved  on  the  performed  simulation  cases. 


CFD  time 
step  [s] 

Required  wall  time  for 
CFD  time  step  [s] 

#  of  CFD  steps  between  data 
exchange  with  GENHEL 

Case  1 

0.005 

0.075 

2 

Case  2 

0.02 

0.019 

1 

Case  3 

0.02 

0.02 

1 

Significance  of  Results 

A  set  of  simulations  was  conducted  to  investigate  the  first  steps  towards  the  practical  real-time  virtual 
dynamics  interface  simulations.  Improvements  to  the  coupling  interface  are  presented  that,  for  a 
simplified  shedding  wake  example,  we  are  able  to  achieve  real-time  execution  speeds  for  the  fully 
coupled  simulation  on  a  Linux  cluster  utilizing  256  processor  cores.  A  demonstration  of  our  first  Pilot- 
in-the-Loop  CFD  (PILCFD)  simulation  for  this  simplified  CFD  wake  case  was  performed  and  results  of 
the  simulation  were  presented. 

Currently,  a  pilot-in-the-loop  CFD  simulation  framework  was  demonstrated  for  a  simplified  CFD  wake 
case  to  establish  the  coupling  interface  between  the  helicopter  flight  dynamics  and  the  CFD  solver 
within  a  real-time  simulation  environment.  In  this  demonstration,  in  order  for  the  CFD  solver  to  perform 
at  real-time  execution  speeds  on  an  available  Linux  cluster,  the  CFD  case  was  run  on  a  small  (-300K 
cells)  grid.  The  small  grid  sized  limited  the  geometry  of  the  wake  generating  structure  to  a  simplified 
3D  backstep.  Additionally,  as  shown  by  the  comparisons  of  Case  1  and  Case  2  results,  significant  wake 
shedding  physics  were  not  resolved  due  to  the  coarse  grid  and  inviscid  assumption  that  were  required  to 
reduce  the  computational  cost  of  the  solver  for  the  demonstration. 

Plans  and  upcoming  events  for  next  reporting  period 

We  will  continue  to  collaborate  with  CRAFT  Tech  to  improve  the  efficiency  of  the  coupling  interface. 
To  achieve  a  practical  PILCFD  simulation  framework  that  can  be  used  in  piloted  virtual  dynamic 
interface  flight  simulation,  future  efforts  will  be  focused  on  improving  the  performance  of  the  CFD 
computations  so  that  a  ship  airwake  model  on  the  order  of  5-10  million  grid  cells  (similar  to  grid  sizes  of 
airwake  datasets  currently  employed  in  dynamic  interface  CFD  simulations)  can  be  included  in  the  fully 
coupled  simulation  framework.  This  will  require  significant  speed  up  of  the  CFD  solver  and  will 


necessitate  hardware  solutions  such  as  the  application  of  GPU  technology  and  ongoing  efforts  to  apply 
the  latest  HPC  architecture  to  increase  performance. 

Efforts  towards  implementing  the  tail  rotor  as  an  actuator  disk  in  the  CFD  domain  are  still  ongoing  and 
we  are  expecting  to  finalize  this  implementation  before  the  next  quarter.  Results  will  be  presented  on  the 
next  quarterly  progress  report. 

There  is  an  ongoing  effort  towards  developing  efficient  methods  to  visualize  the  CFD  flow  field  during 
the  piloted  simulation  will  be  beneficial  to  provide  feedback  of  airwake  dynamics  to  the  pilot. 
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